Skip to content

feat(orchestrator): in-memory queue + disk persistence (QueueStore)#607

Open
gewenyu99 wants to merge 1 commit into
experiment/orchestrator-01-bootstrap-gatingfrom
experiment/orchestrator-03-queue-persistence
Open

feat(orchestrator): in-memory queue + disk persistence (QueueStore)#607
gewenyu99 wants to merge 1 commit into
experiment/orchestrator-01-bootstrap-gatingfrom
experiment/orchestrator-03-queue-persistence

Conversation

@gewenyu99

@gewenyu99 gewenyu99 commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

In-memory task queue for the orchestrator. Synchronous and single-owner, no locking. Tasks carry type, dependencies, attempts, and a structured handoff for the next agent. Every transition rewrites .posthog-wizard/queue.json, the run's log today and the resume point later. nextRunnable returns every pending task whose dependencies are satisfied. Parallelism is decided by the task graph, not by a queue knob.

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci basic-integration
  • /wizard-ci error-tracking-upload-source-maps
  • /wizard-ci misc
  • /wizard-ci revenue

Test an individual app:

  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
Show more apps
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci error-tracking-upload-source-maps/android
  • /wizard-ci error-tracking-upload-source-maps/flutter
  • /wizard-ci error-tracking-upload-source-maps/ios
  • /wizard-ci error-tracking-upload-source-maps/next
  • /wizard-ci error-tracking-upload-source-maps/next-no-posthog
  • /wizard-ci error-tracking-upload-source-maps/node-raw
  • /wizard-ci error-tracking-upload-source-maps/node-rollup
  • /wizard-ci error-tracking-upload-source-maps/node-rollup-typescript-plugin
  • /wizard-ci error-tracking-upload-source-maps/node-webpack
  • /wizard-ci error-tracking-upload-source-maps/nuxt-3-6
  • /wizard-ci error-tracking-upload-source-maps/nuxt-4-3
  • /wizard-ci error-tracking-upload-source-maps/react-native
  • /wizard-ci error-tracking-upload-source-maps/react-vite
  • /wizard-ci error-tracking-upload-source-maps/rust
  • /wizard-ci misc/quack-quack
  • /wizard-ci revenue/stripe

Results will be posted here when complete.

gewenyu99 commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gewenyu99 gewenyu99 force-pushed the experiment/orchestrator-01-bootstrap-gating branch from 7112748 to f6620f8 Compare June 10, 2026 16:10
@gewenyu99 gewenyu99 force-pushed the experiment/orchestrator-03-queue-persistence branch from 48e0be9 to 0b7c4fd Compare June 10, 2026 16:10
@gewenyu99 gewenyu99 marked this pull request as ready for review June 10, 2026 16:13
Comment on lines +146 to +162
enqueue(input: EnqueueInput): QueuedTask {
const task: QueuedTask = {
id: randomUUID(),
type: input.type,
status: 'pending',
dependsOn: input.dependsOn ?? [],
inputs: input.inputs ?? {},
model: input.model,
attempts: 0,
maxAttempts: input.maxAttempts ?? DEFAULT_MAX_ATTEMPTS,
enqueuedBy: input.enqueuedBy ?? 'orchestrator',
createdAt: nowIso(),
};
this.tasks.push(task);
this.reflect();
return task;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing dependency validation allows non-existent task IDs to be added as dependencies, causing tasks to be permanently blocked.

If input.dependsOn contains task IDs that don't exist in the queue, the new task will never become runnable because nextRunnable() checks if all dependencies are in the doneIds set. Non-existent IDs will never be in that set.

This creates a silent deadlock where tasks appear pending but can never run, and isDrained() will return true even though work is incomplete.

enqueue(input: EnqueueInput): QueuedTask {
  // Validate dependencies exist
  if (input.dependsOn) {
    const existingIds = new Set(this.tasks.map(t => t.id));
    const invalid = input.dependsOn.filter(id => !existingIds.has(id));
    if (invalid.length > 0) {
      throw new Error(`Invalid dependencies: ${invalid.join(', ')}`);
    }
  }
  
  const task: QueuedTask = {
    // ... rest of implementation
  };
}
Suggested change
enqueue(input: EnqueueInput): QueuedTask {
const task: QueuedTask = {
id: randomUUID(),
type: input.type,
status: 'pending',
dependsOn: input.dependsOn ?? [],
inputs: input.inputs ?? {},
model: input.model,
attempts: 0,
maxAttempts: input.maxAttempts ?? DEFAULT_MAX_ATTEMPTS,
enqueuedBy: input.enqueuedBy ?? 'orchestrator',
createdAt: nowIso(),
};
this.tasks.push(task);
this.reflect();
return task;
}
enqueue(input: EnqueueInput): QueuedTask {
if (input.dependsOn && input.dependsOn.length > 0) {
const existingIds = new Set(this.tasks.map(t => t.id));
const invalid = input.dependsOn.filter(id => !existingIds.has(id));
if (invalid.length > 0) {
throw new Error(`Invalid dependencies: ${invalid.join(', ')}`);
}
}
const task: QueuedTask = {
id: randomUUID(),
type: input.type,
status: 'pending',
dependsOn: input.dependsOn ?? [],
inputs: input.inputs ?? {},
model: input.model,
attempts: 0,
maxAttempts: input.maxAttempts ?? DEFAULT_MAX_ATTEMPTS,
enqueuedBy: input.enqueuedBy ?? 'orchestrator',
createdAt: nowIso(),
};
this.tasks.push(task);
this.reflect();
return task;
}

Spotted by Graphite

Fix in Graphite


Is this helpful? React 👍 or 👎 to let us know.


export type TaskStatus =
| 'pending'
| 'in_progress'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SNAKE CASE NOOOOO

Comment on lines +70 to +72
function nowIso(): string {
return new Date().toISOString();
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WAT

id: string;
type: string;
status: TaskStatus;
dependsOn: string[];

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we preventing circular deps? dependsOn forming a cycle (A→B→A) or pointing at a non-existent id

Comment on lines +104 to +105
.filter((t) => t.status === 'done' || t.status === 'skipped')
.map((t) => t.id),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about failed? does the queue just stop?

/** The structured handoff the task reported on completion. */
handoff?: TaskHandoff;
/** 'orchestrator' for seeded tasks, or the id of the task that enqueued this one. */
enqueuedBy: string;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enqueuedBy is free-form and tasks can enqueue tasks. Without a depth or count bound, a misbehaving task type can grow the queue unboundedly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants